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Background / Context: 

Description of prior research and its intellectual context. 

Recent scholarship on school refonn has highlighted the need to use new approaches to enact 
change and recognize the critical interplay between refonn practices, the process of 
implementation, and local context (Penuel, Fishman, Cheng, & Sabelli, 201 1). One such 
approach, improvement science, focuses on using continuous improvement processes to address 
problems of practice in ways that contribute to organizational learning (Bryk, Gomez, Grunow, 

& LeMahieu, 2015). Key to improvement science is the creation of networked improvement 
communities that engage in disciplined cycles of inquiry where data is used to understand the 
problem and test potential solutions (Bryk, Gomez, Grunow, & Hallinan, 2011). 

Using data for school improvement is not new. Indeed, data-driven decision-making has been a 
priority for 20 years (DeBray-Pelot & McGuinn, 2009). Yet, teacher data use is typically 
characterized as guiding teachers to make well-informed instructional decisions (Hamilton et ah, 
2009). Even when teachers are able to use data in ways that benefit their instruction, the data is 
usually comprised of data directly from the district’s accountability systems. In other words, the 
data teachers are asked to draw on for improvement purposes may be used by administrators or 
central office staff to hold them accountable. 

Improvement science notes a distinction between data for accountability and data for 
improvement, with different data needs and social conditions of use (Yeager, Bryk, Muhich, 
Hausman, & Morales, n.d.). While data for accountability focuses on outcomes to identify 
whether expectations are being met, data for improvement is more practical, providing evidence 
about the practice itself (Yeager et ah, n.d.). This broader focus on evidence better captures the 
ways in which educator decision-making is not solely based on data, but a range of evidence. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

Our focus is on the process educators use to interpret data to turn it into usable knowledge 
(Honig & Coburn, 2008) while engaging in a continuous improvement process. We examine the 
types of evidence educators draw upon, its perceived relevance, and the social context in which 
the evidence is examined. Evidence includes data related to student or school outputs (e.g., 
passing rates), inputs (e.g., student demographic infonnation), school processes (e.g., data on 
program implementation) and perceptions (e.g., surveys) (Marsh, 2012). Previous research also 
indicates that educators draw on evidence substitutes — anecdotal information, experience, 
intuition, ideological preference, and customary practice (Bryk & Gomez, 2008; Dorman & 
Binnewies, 2015; Ingram, Louis, & Schroeder, 2004). Considering this diversity of evidence 
enables us to examine how teacher-led implementation teams come to shared conclusions about 
what they are learning about their improvement efforts. We ask the following research questions: 

(1) How are educators learning about and using data for the improvement of a school- 
wide refonn? 

(2) What types of evidence are they citing to make claims about what they are learning? 
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(3) What are the barriers educators face as they learn how to apply data to make claims 
about what they are learning? 


Setting: 

Description of the research location. 

The data used in this paper were collected as part of a larger study to explore new approaches of 
scaling effective practices in high schools. At the time of the study, the district served 
approximately 80,000 students, the majority of whom are low-income or from traditionally 
underserved racial/ethnic groups. (Please see Table 1). Demographic characteristics are rounded 
to the nearest 20 percent to prevent the use of identifying school characteristics. Desert Grove, 
Forest Glen, and Valley High Schools all serve high concentrations of low-income students and 
students of color. Pseudonymous names are used for participant confidentiality. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The participants in this study are members of school development teams. The development team 
at each high school is comprised of seven to eight members. Teams are teacher-led, with two 
administrators participating on the team at Valley High School. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

This program is based on a continuous improvement approach to implementation. The school 
development teams were tasked with taking a set of practices designed to improve student 
ownership and responsibility (SOAR) and align them with their school context. The focus on 
student ownership and responsibility came our of a year-long mixed methods study of effective 
practices in higher value-added high schools in the district. To identify the specific practices to 
implement in their school, teams were trained in the use of Plan, Do, Study, Act (PDSA) as a 
tool to guide their improvement efforts. This process involved identifying an element of the 
design they wanted to implement, collecting data on the success of this component, reviewing 
this data on how to improve this component of the innovation design. Learning was then shared 
across schools at quarterly meetings, as an opportunity to leam from one another. 

Although each school implemented practices in somewhat distinct ways, four central practices 
are common across schools to help schools reach their program goal of creating student 
ownership and responsibility (SOAR). These components were all taught during advisory 
periods. First, teachers taught students about growth mindsets. Second, teachers adopted 
practices that helped students monitor their grades. Third, students set goals of how they would 
reach academic goals. Finally, teachers worked with students to create problem-solving 
strategies for how they would reach these goals. 

Research Design: 

Description of the research design. 
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A qualitative case study design (Lincoln & Guba, 1985) allows us to probe how school 
stakeholders, particularly the teacher-led teams, used evidence in the ongoing development of the 
innovation. School development teams met regularly over two years in a networked fashion, with 
each school-based team having time to plan and engage in PDSA cycles, while also facilitating 
sharing across schools. In the second year, each school began full implementation, with the 
school development team leading efforts in their school. We observed and participated in 
meetings of the school development teams and conducted interviews with the team members and 
others in their schools at several points in the process. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

The data used for this analysis come from two sources. We refer to the primary data source as 
the “process data” and it includes observational and artifact data from district-wide meetings of 
the school development teams, including observational fieldnotes and audio recordings of all 
design team meetings, artifacts distributed or produced at the meetings, and feedback fonns 
completed by members after each meeting. Research team members collected these data sources 
by attending regular meetings with the school implementation teams (i.e., teacher-led teams) and 
program developers from Spring 2013 to Spring 2015. In addition to this process data, we also 
conducted fieldwork visits in each of the partner schools to monitor implementation. Teams of 
three researchers spent four days in each of the schools on each of these visits. We draw on data 
from 45 semi-structured interviews with implementation team members, where they were asked 
about their experiences with data use and the continuous improvement process (see Table 2). 

The process data was analyzed for the design, development, and implementation phases, 
respectively, to produce summary memos that describe evidence from that phase according to a 
framework drawn from the project’s framework for innovation design and development. This 
framework consisted of several a priori codes in addition to codes that emerged inductively from 
the data. Following each fieldwork visit, the research team coded transcripts based on a coding 
schema related to the project’s framework for quality implementation. The coding process was 
iterative in nature with members of the research team comparing coding to ensure a consistent 
understanding and application of codes (Corbin & Strauss, 2008). The team then wrote a 
summary memo for each field visit, probing similarities and differences across the three partner 
schools. For this paper, we draw on analytic codes related to the implementation teams’ use of 
evidence, specifically in the context of the continuous improvement of the innovation design. 
From the analysis of the school-based fieldwork, we draw on codes for continuous improvement, 
the capacity of implementation team members, and external support to the implementation team. 
From the process analysis, we draw on codes for piloting and PDSA, participant understanding 
of the continuous improvement process, and the integration of the research team into the process. 


Findings / Results: 

Description of the main findings with specific details. 

At the conclusion of the design process, participants were tasked with co-creating measures 
related to intermediary and proximal outcomes of the SOAR innovation with the research team 
and program developers. Participants resisted this task, expressing concern that the innovation 
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was being measured by a data point rather than trying to understand meaningful changes to 
teacher practice. Following the suggestion that they could monitor the progress of 
implementation with the use of non-punitive checklists and learning walks, most implementation 
team members were concerned that their district had the tendency to misuse such data. 
Participants associated data as being used for accountability purposes, and that to quantify 
outcomes of the innovation would cause their work to fall into the same trap. 

It was not until late in the development process that school implementation teams’ resistance to 
data began to subside, and they identified measures that they felt were appropriate. These data 
were quantitative data that was already available within the district and included failure rates, 
attendance and tardy rates, school infraction rates, the number of incomplete assignments, 
student use of credit recovery programs, and continuation rates. In the process, they often 
diminished the role that perceptual data could play in informing their collective learning. 

Yet, team members were most likely to draw on perceptual data and other evidence substitutes to 
understand the extent to which the SOAR innovation was having an impact on students in their 
schools. School teams often had perceptual evidence from surveys or discussions they had with 
colleagues in professional development that formed the basis of their collective learning. Still, 
when sharing evidence of learning across schools, teams focused on evidence from the district’s 
administrative data, often not sharing other evidence that they had actually drawn upon to inform 
their learning. For instance, teachers at Forest Glen collected student survey data, but they did 
not frame it as data. This shift reflects how the teams came to leam to value and use evidence for 
improving the innovation, but still felt constrained by the context of the district when sharing 
with those outside their team. The result was that implementation team members drew on 
different sets of evidence to inform their internal learning (perceptual evidence) and the types of 
evidence that they shared when they presented their learning to other stakeholders in the district. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

Teacher evidence use is often portrayed as drawing on a mix of evidence as well as “gut 
feelings” about what would work best for one’s students. In this paper, we describe how teachers 
use data for school improvement and how this changed through the development of an 
innovation to increase student ownership and responsibility. We find evidence that teachers are 
likely to draw on a mix of evidence as well as evidence substitutes when refining the innovation 
at the school-level. While implementation teams considered outcome data, they gravitated 
towards perceptual data. They used this data to make small improvements to the innovation but 
also to gauge the level of teacher buy-in and make modifications that would better meet teachers’ 
needs. As continuous improvement continues to expand as an approach to reform within schools, 
it is important for scholars to understand the particular contexts in which practitioners are asked 
to consider evidence. Though resistance to data use and evidence-driven practices amongst 
teachers decreased over time, and they even came to value data use to inform and drive 
improvement efforts, they continued to struggle as they had to straddle the overarching context 
of accountability, despite their willingness to see data use through the lens of continuous 
improvement. The context of accountability is ubiquitous and its impact on continuous 
improvement efforts is one that must be better understood so that the tension between two 
differing ways of considering data can be mitigated. 
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Table 1. Demographic Profile of Partner District and Schools 



All 

high schools 

Desert Grove 
High School 

Forest Glen 

High School 

Valley 

High School 

Student enrollment 

20,504 

>1500 

700-1200 

>1500 

Student race/ethnicity 

Percent Hispanic 

59% 

40-60% 

>80% 

>80% 

Percent African 
American 

25% 

20-40% 

<20% 

<20% 

Percent White 

8% 

20-40% 

<20% 

<20% 

Percent 

economically 

disadvantaged 

70% 

40-60% 

>80% 

>80% 

Percent LEP 

6% 

<5% 

>5% 

>5% 

Teacher race/ethnicity 

Percent Hispanic 

17% 

<20% 

20-40% 

20-40% 

Percent African 
American 

29% 

<20% 

<20% 

<20% 

Yrs. teaching 
experience 

10.9 

10-12 

10-12 

8-10 


Source. District administrative data, 2012-2013 school year. 
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Table 2. Data Sources 


Data Sources 

Phase 1- 
Design 

Phase 2- 
Development 

Phase 3- 
Implementation 

Total 

Data collection 
dates 

January 20 13 — 
June 2014 

August 2013 — 
June 2014 

August 2014 - 
June 2015 

34 months 

Audio Files 

62 h 22 m 

66 h 21 m 

61 h 50 m 

186 h 33 m 

Field Note Logs 

28 

24 

19 

71 

Distributed 

Artifacts 

147 

236 

74 

457 

Research 

Reflection Fonns 

6 

5 

5 

16 

Participant 

Feedback Forms 

97 

141 

82 

320 

Interview 

Transcripts 

24 

23 

0 

47 

Fieldwork Visit 



October 

2014 

April 

2015 


Implementation 
team members 



24 

21 

45 
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